Storage medium having fingernail image processing, information processing apparatus, and control method of information processing apparatus

ABSTRACT

An information processing apparatus detects a printing-target fingernail area from a first captured image, creates print data for performing printing in the detected fingernail area, and detects a printing-target fingernail area from a second captured image before transmitting the print data to a printer. The print data is corrected based on a difference between the first and second detection results. In addition, a change in position, gradient, and shape of a fingernail based on the first detection results and the second detection results is detected, and it is determined whether one of an amount of change in position, an amount of change in gradient, and an amount of change in shape of a fingernail is larger than or equal to a predetermined threshold value.

BACKGROUND OF THE INVENTION Field of the Invention

The present disclosure relates to a technique of image processing forprinting a nail art on a fingernail.

Description of the Related Art

In recent years, it has been made possible to print nail art on afingernail by using a printer. In the following, a printer for printingnail art on a fingernail is referred to as a nail printer.

Japanese Patent Laid-Open No. 2016-32773 has disclosed a technique toresume printing, in a case where a change in the fingernail position isdetected during the printing operation on the fingernail in a nailprinter, by stopping the printing operation of the print head andadjusting the resumption position of printing after the change in thefingernail position stops.

SUMMARY OF THE INVENTION

However, in Japanese Patent Laid-Open No. 2016-32773, it is possible todeal with a case where the fingernail moves during printing, but in acase where the fingernail moves before printing, that is, in a casewhere the position or the like of the fingernail changes during theperiod from the detection of the printing-target fingernail area untilprinting is performed, it is not possible to print nail art correctly.

Consequently, in view of the above-described problem, an object of oneembodiment of the present invention is to make it possible to performprinting on a fingernail correctly.

One embodiment of the present invention is a non-transitory computerreadable storage medium storing a program for causing a computer of aninformation processing apparatus to function as: a first detection unitconfigured to detect a printing-target fingernail area from a firstcaptured image; a creation unit configured to create print data forperforming printing in a fingernail area detected by the first detectionunit; a second detection unit configured to detect a printing-targetfingernail area from a second captured image before transmitting theprint data to a printer; and a correction unit configured to correct theprint data based on a difference between first detection results by thefirst detection unit and second detection results by the seconddetection unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A and FIG. 1B are outline diagrams showing a configuration of asystem;

FIG. 2 is a diagram showing a nail art selection screen;

FIG. 3 is a diagram showing a print data creation screen;

FIG. 4 is a diagram explaining an area setting mode that sets aprintable area;

FIG. 5 is a diagram showing a nail image data setting screen;

FIG. 6 is a function block diagram of an information processingapparatus;

FIG. 7A and FIG. 7B indicate a sequence diagram showing a flow ofprocessing;

FIG. 8A to FIG. 8C are diagrams explaining correction processing ofprint data; and

FIG. 9 is a flowchart of correction processing of print data.

DESCRIPTION OF THE EMBODIMENTS

In the following, embodiments of the present disclosure are explained indetail. The following embodiments are illustrations for explaining thepresent disclosure and not intended to limit the present disclosure onlyto those embodiments. Further, it is possible to modify the presentdisclosure in a variety of manners as long as the modification does notdepart from the gist thereof.

First Embodiment

<System Configuration>

A system in the present embodiment has an information processingapparatus and a printer. In the present embodiment, explanation is givenby taking a tablet terminal as an example of the information processingapparatus. However, the information processing apparatus is not limitedto the tablet terminal. As the information processing apparatus, it ispossible to apply various devices, such as a mobile terminal, a note PC,a smartphone, a PDA (Personal Digital Assistant), and a digital camera.Further, in the present embodiment, as the printer, it is possible toapply, for example, an ink jet printer, a 3D printer and the like. Theprinter of the present embodiment may be a multifunction printer havinga plurality of functions, such as the copy function, the FAX function,and the print function. The printer of the present embodiment has afunction of performing drawing directly on a fingernail of a human hand.In the present embodiment, explanation is given by distinguishing theinformation processing apparatus and the printer as separateapparatuses, but an aspect may be accepted in which an apparatus havingthe functions of both apparatuses in an integrated manner is used.

FIG. 1A and FIG. 1B are diagrams explaining the system having aninformation processing apparatus 101 and a printer 151 of the presentembodiment. In FIG. 1A, a block diagram of the information processingapparatus 101 and the printer 151 is shown. In FIG. 1B, a schematicdiagram of an outer appearance of the printer 151 is shown. In thefollowing, by using FIG. 1A and FIG. 1B, the configuration of theinformation processing apparatus 101 and the printer 151 is explained.

<Information Processing Apparatus>

As shown in FIG. 1A, the information processing apparatus 101 has aninput interface 102, a CPU 103, a ROM 104, a RAM 105, an externalstorage device 106, an output interface 107, a communication unit 109,and a GPU 110. These are connected to one another via a system bus.

The input interface 102 is an interface for receiving a data input andoperation instructions from a user via an operation unit (not shownschematically) including a physical keyboard, buttons, a touch panel andthe like. The aspect of the information processing apparatus 101 in thepresent embodiment is such that a display unit 108, to be describedlater, and at least part of the operation unit are integrated into oneunit and for example, the output of a screen and the reception of anoperation from a user are performed on the same screen.

The CPU 103 is a system control unit and controls the entire informationprocessing apparatus 101, such as the execution of programs andactivation of hardware. In the ROM 104, control programs executed by theCPU 103, data tables, a built-in operating system (in the following,referred to as OS), and data of programs and the like are stored. In thepresent embodiment, each control program stored in the ROM 104 performssoftware execution control, for example, such as scheduling, taskswitch, and interrupt processing, under the management of the built-inOS stored in the ROM 104.

The RAM 105 includes an SRAM (Static Random Access Memory), a DRAM(Dynamic Random Access Memory) or the like. In the RAM 105, data may bestored by a primary battery for data backup, not shown schematically. Inthat case, it is possible to store data of program control variables andthe like in the RAM 105 without volatilizing the data. Further, in theRAM 105, a memory area is also provided, which stores settinginformation on the information processing apparatus 101, management dataon the information processing apparatus 101, and the like. Furthermore,the RAM 105 is also used as a main memory and a work memory of the CPU103.

In the external storage device 106, an application that provides aprinting execution function, a printing information generation programthat generates printing information that the printer 151 can interpret,and the like are stored. Further, in the external storage device 106,various programs, such as an information transmission and receptioncontrol program that performs transmission and reception of informationwith the printer 151 connected via the communication unit 109, andvarious kinds of information that these programs use, and the like arestored.

The output interface 107 is an interface that performs control of thedisplay of data by the display unit 108, the notification of the stateof the information processing apparatus 101, and the like.

The display unit 108 has an LED (Light-Emitting Diode), an LCD (LiquidCrystal Display) or the like and performs the display of data, thenotification of the state of the information processing apparatus 101,and the like. It may also be possible to receive an input from a uservia the display unit 108 by installing a soft keyboard having keys, suchas a numerical value input key, a mode setting key, a determination key,a cancellation key, and a power source key, on the display unit 108.Further, the display unit 108 may be configured as a touch panel displayas described previously. The display unit 108 is connected with thesystem bus of the information processing apparatus 101 through theoutput interface 107.

The communication unit 109 is configured so as to perform datacommunication by connecting with an external device, such as the printer151. It is possible for the communication unit 109 to connect to anaccess point (not shown schematically) within the printer 151. That is,in the present embodiment, it is possible for a communication unit 156within the printer 151 to operate as an access point. The access pointis an example and what is required is that the communication unit 156operates as a master station at the time of performing wirelesscommunication in conformity to the IEEE 802.11 series, and for example,the communication unit 156 may operate as a group owner of Wi-Fi Direct.By the communication unit 109 connecting to the access point within theprinter, it is made possible for the information processing apparatus101 and the printer 151 to perform wireless communication with eachother. It may also be possible for the communication unit 109 to performdirect communication with the printer 151 by wireless communication orto communicate with the printer 151 via an external access point (accesspoint 131) existing outside. As the wireless communication method,mention is made of, for example, Wi-Fi (Wireless Fidelity) (registeredtrademark), Bluetooth (registered trademark) and the like. Further, asthe access point 131, mention is made of, for example, a device such asa wireless LAN router. In the present embodiment, the method in whichthe information processing apparatus 101 and the printer 151 connectdirectly to each other without the intervention of the external accesspoint 131 is called the direct connection method. Further, the method inwhich the information processing apparatus 101 and the printer 151connect to each other via the external access point 131 is called theinfrastructure connection method. The aspect may also be one in whichthe information processing apparatus 101 and the printer 151 areconnected via a wire, such as a USB cable.

It is possible for the GPU 110 to perform an efficient calculation byperforming parallel processing of more data, and therefore, in a casewhere a learning model, such as deep learning, is processed, it iseffective to perform the processing with the CPU 110. Consequently, inthe present embodiment, the GPU 110 is used relating to the processingusing a learning model, in addition to the CPU 103. Specifically, in acase where a learning program including a learning model is performed,the learning model is processed by the CPU 103 and the GPU 110performing the calculation in cooperation with each other. In theprocessing using a learning model, the calculation may be performed byonly one of the CPU 103 and the GPU 110.

In the present embodiment, it is assumed that the information processingapparatus 101 stores a predetermined application in the ROM 104, theexternal storage device 106 or the like. The predetermined applicationis an application program for transmitting a print job for printing nailart data to the printer 151 in response to, for example, the operationfrom a user. The application that has the function such as this iscalled a nail application hereinafter. The nail application may haveanother function in addition to the print function. For example, thenail application in the present embodiment may have a function toactivate a camera of an image capturing unit 157 of the printer 151 bycommunicating with the printer 151. That is, the nail application mayhave a function to transmit a camera activation job to the printer 151,and the like, other than the print job. Further, the predeterminedapplication stored in the ROM 104, the external storage device 106 orthe like is not limited to the nail application and may be anapplication program having a function other than printing.

<Printer>

The printer 151 has a ROM 152, a RAM 153, a CPU 154, a print engine 155,the communication unit 156, and the image capturing unit 157. Thesecomponents are connected to one another via a system bus. As shown inFIG. 1B, in the printer 151, a printing target insertion unit 158 thatis a space for inserting a printing target is provided. FIG. 1B showsthe way a user inserts his/her own hand into the printing targetinsertion unit 158. As described above, in the present embodiment, it issupposed that a human hand is inserted into the printing targetinsertion unit 158. Further, in the present embodiment, the printingtarget is a fingernail. In the printing target insertion unit 158, fourrails for placing fingers are mounted and it is possible to insert oneto four fingers. The four rails are designed so as to be capable oflifting and lowering. Consequently, by the rail lifting before printing,it is possible to adjust the height of a fingernail at the heightoptimum for printing. This lifting and lowering operation is performedby instructions from the nail application. In the present specification,an example is shown in which the hand fingernail is the printing target,but it is also possible to apply the present embodiment also to a casewhere printing is performed on a nail of a foot, not limited to a hand.

In the ROM 152, control programs executed by the CPU 154, a data table,and data of the OS program and the like are stored. In the presentembodiment, each control program stored in the ROM 152 performs softwareexecution control, such as scheduling, task switch, and interruptprocessing, under the management of the built-in OS stored in the ROM152.

The RAM 153 includes an SRAM, a DRAM or the like. In the RAM 153, datamay be stored by a primary battery for data backup, not shownschematically. In this case, it is possible to store data of programcontrol variables and the like in the RAM 105 without volatilizing thedata. Further, a memory area that stores setting information on theprinter 151, management data on the printer 151, and the like is alsoprovided in the RAM 153. Furthermore, the RAM 153 is also used as a mainmemory and a work memory of the CPU 154 and it is possible for the RAM153 to temporarily store printing information received from theinformation processing apparatus 101. various kinds of information andthe like.

The CPU 154 is a system control unit and controls the entire printer 151by executing programs and activating hardware. The print engine 155forms an image on a printing-target medium, such as a fingernail, whichis inserted into the printing target insertion unit 158 by using aprinting material, such as ink, based on the information stored in theRAM 153 or the print job received from the information processingapparatus 101.

It is possible for the communication unit 156 to operate as an accesspoint for performing wireless communication by the direct connectionmethod with an external apparatus, such as the information processingapparatus 101. It is possible for this communication unit 156 to connectwith the communication unit 109 of the information processing apparatus101. It may also be possible for the communication unit 156 to directlycommunicate with the information processing apparatus 101 by wirelesscommunication or communicate with the information processing apparatus101 via the external access point 131. In a case where the communicationunit 156 connects with the external access point 131 by theinfrastructure connection method, the communication unit 156 operates asa slave station and the external access point 131 operates as a masterstation. Further, the communication unit 156 may have hardware thatfunctions as an access point or the communication unit 156 may operateas an access point by software for causing the communication unit 156 tofunction as an access point.

The image capturing unit 157 is a device having an image capturingfunction and belongs to the printer 151 and is installed therein. Theimage capturing unit 157 has a function to capture a predetermined areaincluding a printing target (specifically, fingernail) that is insertedinto the printing target insertion unit 158 and send the captured image(still image, moving image and the like) to the information processingapparatus 101 real time. In the present embodiment, the image capturingunit 157 captures a moving image and a still image and is a cameramodule having at least a lens and an image sensor. The lens collectslight rays from the printing target that is inserted into the printingtarget insertion unit 158 and forms an image on the image sensor. Theimage sensor converts the light rays collected by the lens into anelectrical signal that the CPU 154 can process. It may also be possibleto use a smartphone, a mobile terminal, a digital camera or the like inplace of the camera module as a device having the image capturingfunction in a case where the device has the function such as this. Theprint engine 155 performs printing for the printing target that isinserted into the printing target insertion unit 158.

It may also be possible to attach a memory, such as an external HDD oran SD card, to the printer 151 and the information stored in the printer151 may be stored in the memory. Further, the configuration shown inFIG. 1A and FIG. 1B is merely exemplary and each of the informationprocessing apparatus 101 and the printer 151 may have components otherthan those described previously, but explanation thereof is omittedhere.

Definition of Terms

Next, terms that are used in the present embodiment are explained. Thepresent embodiment is an aspect in which nail art is printed mainly on afingernail. Further, the present embodiment is an aspect in which nailart is printed on each fingernail of one hand. In general, the nail artthat is printed on each fingernail has the same concept, but there is acase where the nail art that is printed on each individual fingernail isnot exactly the same nail art. For example, in a nail art set of designA, ten nail arts (nail arts corresponding to each fingernail of tenfingers) are included and the ten nail arts have the common designconcept, but there is a case where the patterns are not exactly thesame. In view of the above-described points, in the present embodiment,terms are defined as follows.

-   -   “nail image data”: refers to image data of a nail art that is        printed on one fingernail.    -   “nail art data”: refers to an aggregate of a plurality of pieces        of nail image data. That is, the nail art data can also be said        as a dataset of a plurality of pieces of nail image data.        Typically, the nail art data is image data aggregating images of        each piece of nail image data corresponding to the nail arts of        ten fingernails of the right hand and the left hand together.        The nail art data may be data aggregating each of ten pieces of        nail image data (that is, a set of ten pieces of image data) or        may be image data obtained by combining each of ten pieces of        nail image data into one image.

As described above, it is assumed that in a case where “nail image data”is referred to, the data of the image of the nail art of each individualfingernail is referred to, and in a case where “nail art data” isreferred to, data of a set of images of ten nail arts is referred to.

<Outline of Printing of Nail Art>

In the present embodiment, the nail application is activated by the CPU103 of the information processing apparatus 101 executing the program ofthe nail application stored in the ROM 104 or the external storagedevice 106. Then, by using the nail application, it is possible for auser to print a nail art on a fingernail by reflecting the nail imagedata included in the nail art data selected by the user him/herself inthe printable area. That is, the following series of operations isperformed by using the nail application. (1) A user selects a set ofnail image data (that is, nail art data) on the application. (2) Theapplication activates the camera located inside the nail printer. (3)The application displays the camera image transmitted from the nailprinter. (4) The user inserts his/her hand into the nail printer. (5)The application detects the area of the fingernail from the camera imageand sets the area of the fingernail on which the nail art is printed.(6) The application reflects the nail image data included in the nailart data in the set printable area. (7) The user edits the printablearea in a case of adjusting the set printable area. (8) The applicationcauses the nail printer to perform printing by using the reflected imagedata. A user who inserts his/her hand into the printer 151 and a userwho operates the application may be the same user or different users.

<Graphical User Interface of Nail Application>

In order to make understanding easy, a graphical user interface (in thefollowing, described as GUI) screen that is displayed in the nailapplication is explained first. The GUI screen that is explained in thefollowing is displayed on the display unit 108 by the CPU 103 thatexecutes the nail application. Further, explanation is given on theassumption that the input interface 102 functions as the operation unitintegrated with the display unit 108 into one unit. The GUI screen ofthe nail application in the present embodiment is roughly divided intothree kinds (referred to as first to third GUI screens). The first GUIscreen is a nail art data selection screen shown in FIG. 2. The secondGUI screen is a nail image data setting screen shown in FIG. 5. Thethird GUI screen is a print data creation screen shown in FIG. 3. In thefollowing, by using each screen, the GUI screens of the nail applicationare explained.

FIG. 2 is a diagram showing a nail art data selection screen 201 as anexample of the first GUI screen. As shown in FIG. 2, the nail art dataselection screen 201 has a nail art data display area 202 and aDetermine button 204.

In the nail art data display area 202, a plurality of pieces of nail artdata corresponding to each design concept of a nail art that is printedon each fingernail is displayed. Specifically, on the screen in FIG. 2,four pieces of nail art data 203 are displayed. In one piece of the nailart data 203, ten kinds of nail image data corresponding to each of thefingernails of the thumbs, the index fingers, the middle fingers, thethird fingers, and the little fingers of the left hand and the righthand are included. It is possible for a user to select one piece or aplurality of pieces of the nail art data 203 and also cancel theselected nail art data 203. As an example of the method in which a userselects or cancels the nail art data 203, it is possible for a user toselect the nail art data 203 by tapping the nail art data 203 once.Further, it is possible for a user to cancel the selected state bytapping the selected nail art data 203 once again.

In a case where a user presses down the Determine button 204 afterselecting one piece or a plurality of pieces of the nail art data 203from the nail art data display area 202, a nail image data settingscreen 501 shown in FIG. 5 is displayed in place of the nail art dataselection screen 201.

In the present embodiment, explanation is given on the assumption thatin a case where various buttons are operated, the operation is referredto as “press down” operation and in a case where each area is operated,the operation is referred to as “tap” operation, “touch” operation,“pinch-in” operation, “pinch-out” operation and the like as theoperation of the touch panel, but these are merely exemplary. Forexample, the press down operation of various buttons may be implementedby touching various buttons on the touch panel. Further, for example,each area may be operated by the cursor operation using a mouse or thelike. Furthermore, the input interface 102 may have variousdirection-indicating buttons and the operation of each area may beperformed by using the direction-indicating buttons.

FIG. 5 is a diagram showing the nail image data setting screen 501 as anexample of the second GUI screen. As shown in FIG. 5, the nail imagedata setting screen 501 has a nail art data display area 502, a lefthand preview area 503, a right hand preview area 504, a selection modechangeover switch 505, a selection cancellation button 506, and asetting completion button 507.

The nail art data display area 502 is an area that displays the nail artdata selected by a user on the nail art data selection screen 201.

In the left hand preview area 503, a total of five fingernail areas 508of the thumb to the little finger of the left hand are included and inthe right hand preview area 504, a total of the five fingernail areas508 of the thumb to the little finger of the right hand are included. Auser selects nail art data or nail image data from the nail art datadisplay area 502. Due to this, it is possible to reflect the selectednail art data or nail image data in the fingernail area 508 in the lefthand preview area 503 and the right hand preview area 504.

The selection mode changeover switch 505 is a switch to change over themethods of selecting nail art data or nail image data that is reflectedin the fingernail area 508 included in the left hand preview area 503and the right hand preview area 504. In a case where a batch selectionmode is ON, it is possible to reflect at a time the nail art dataselected by a user from the nail art data display area 502 in thefingernail areas 508 included in the left hand preview area 503 and theright hand preview area 504, respectively. In a case where the batchselection mode is OFF, it is possible to reflect the nail image dataselected by a user from the nail art data display area 502 in one of thefingernail areas 508 included in the left hand preview area 503 or theright hand preview area 504.

The selection cancellation button 506 is a button that can be selectedin a case where nail image data is already reflected in at least one ofthe fingernail areas 508 included in the left hand preview area 503 andthe right hand preview area 504. In a case where a user presses down theselection cancellation button 506, the selection of all reflected nailimage data is cancelled and the state returns to the state where no nailimage data is selected.

The setting completion button 507 is a button that can be selected in acase where nail image data is reflected in one or more of the fingernailareas 508 included in the left hand preview area 503 and the right handpreview area 504. In a case where a user presses down the settingcompletion button 507, a print data creation screen 301 shown in FIG. 3is displayed in place of the nail image data setting screen 501.

FIG. 3 is a diagram explaining the print data creation screen 301 forcreating print data as an example of the third GUI screen. The printdata creation screen 301 has a setting data display area 302, aprinting-target display area 303, a printer search button 304, a printername display area 305, an Image capture button 305, a preview area 307,a printable area setting button 308, and a print button 310. In thepreview area 307, a printable area 309, to be described later, isincluded.

The setting data display area 302 is an area that displays the nailimage data set on the nail image data setting screen 501 shown in FIG. 5by dividing the nail image data into four areas at most of the data ofthe left hand and the right hand. FIG. 3 shows an aspect, as an exampleof the method of dividing the setting data, in which the data isdisplayed as one combination of the index finger, the middle finger, thethird finger, and the little finger of the left hand, one combination ofthe thumb of the left hand, one combination of the index finger, themiddle finger, the third finger, and the little finger of the righthand, and one combination of the thumb of the right hand. In the settingdata display area 302 in FIG. 3, the display is shown in a case wherethe nail image data is set for all the fingernails of the left hand andthe right hand. As shown schematically, in the setting data display area302, setting data 302(a) of the index finger, the middle finger, thethird finger, and the little finger of the left hand and setting data302(b) of the thumb of the left hand are displayed. Further, in thesetting data display area 302, setting data 302(c) of the index finger,the middle finger, the third finger, and the little finger of the righthand and setting data 302(d) of the thumb of the right hand are alsodisplayed. In each area of the setting data 302(a) to 302(d), four slotsin which nail image data is input are provided.

In the printing-target display area 303, four icons indicating whichfingernail each of the setting data 302(a) to 302(d) corresponds to aredisplayed. That is, as in the setting data display area 302, the fouricons are displayed as one combination of the index finger, the middlefinger, the third finger, and the little finger of the left hand, onecombination of the thumb of the left hand, one combination of the indexfinger, the middle finger, the third finger, and little finger of theright hand, and one combination of the thumb of the right hand. Asdescribed previously, in the printing-target display area 303 shown inFIG. 3, the display is shown in a case where the nail image data is setfor all the fingernails of the left hand and the right hand. An icon303(a) indicates that the setting data 302(a) corresponds to the indexfinger, the middle finger, the third finger, and the little finger ofthe left hand and an icon 303(b) indicates that the setting data 302(b)corresponds to the thumb of the left hand. An icon 303(c) indicates thatthe setting data 302(c) corresponds to the index finger, the middlefinger, the third finger, and the little finger of the right hand and anicon 303(d) indicates that the setting data 302(d) corresponds to thethumb of the right hand.

In a case where the printer search button 304 is pressed down by a user,processing to search for a printer capable of communicating with thenail application is performed. As a result of the search, in a casewhere one or a plurality of printers including the printer 151 isdiscovered, the nail application displays information specifying each ofthe discovered printers. The information specifying the discoveredprinters is displayed in a list and it is possible for a user to selectone arbitrary printer from the list (here, it is assumed that a userselects the printer 151).

The printer name for uniquely identifying the selected printer 151 isdisplayed in the printer name display area 305. It is possible for thenail application to automatically search for the printer and display theprinter list also in a case where the screen transitions to the printdata creation screen 301 in the state where no printer is registered notonly in a case where a user presses down the printer search button 304.

The Image capture button 306 is a button for communicating with theprinter 151 displayed in the printer name display area 305 by beingpressed down by a user and receiving a moving image captured by theimage capturing unit 157 of the printer 151 real time and displaying themoving image in the preview area 307. In the present embodiment, it isassumed that a human hand is inserted into the printing target insertionunit 158. Because of this, for example, by a user inserting his/her onehand and pressing the Image capture button 306 with the other hand, theimage of the top of the user's hand including the fingernails isdisplayed real time in the preview area 307. An image, such as a movingimage, is displayed in the preview area 307 is not necessarily limitedto a case where the Image capture button 306 is pressed down. Forexample, a case is considered where the nail application is capable ofcommunicating with the printer 151 at the time of transition to theprint data creation screen 301 in the state where the printer 151 isalready registered. In this case, it is possible for the nailapplication to receive a moving image captured by the image capturingunit 157 of the printer 151 by automatically communicating with theprinter 151 at the time of transition to the print data creation screen301.

The use described previously is an example and another use can also beconsidered. For example, the GUI screen 501 may have an image capturingbutton and a user of a nail salon inserts his/her one hand and anemployee of the nail salon presses down the image capturing button onthe UI screen 501.

The printable area setting button 308 is a button for moving to the areasetting mode to set the printable area 309 by being pressed down by auser in a case where a moving image is being displayed in the previewarea 307.

In response to the printable area setting button 308 being pressed down,the mode moves to the area setting mode capable of setting the printablearea 309. FIG. 4 is a diagram explaining an example in which thefingernail area is automatically detected from a still image and theprintable area 309 is automatically set as an example of the method ofautomatically detecting a fingernail area.

Specifically, first, a user sets the preview area 307 to the areasetting mode by pressing down the printable area setting button 308.Then, the CPU acquires the still image of the hand including thefingernails of the user from the printer 151, which is acquired by theimage capturing unit 157 performing image capturing, and automaticallysets the printable area 309 by automatically detecting the fingernailarea from the still image. As an example of the automatic detectionmethod, it may also be possible to use a learning model created bymachine learning. The learning model that is used here is a learningmodel that outputs results of detecting the fingernail area in a casewhere a still image including fingernails is input. The nail applicationoutputs the results of detecting the fingernail area from the stillimage. The output fingernail area is automatically set as the printablearea 309 as shown in FIG. 4. Further, the detection of the fingernailarea is performed for each finger and for example, in a case of fourfingers, the fingernail area detection is performed four times. At thetime of displaying the printable area 309, which is detection results,an image (referred to as finger image) for each finger is displayed inthe preview area 307 and for example, in a case of four fingers, fourfinger images are displayed in the preview area 307. The printable area309 is displayed on the finger image and the one printable area 309 isset and displayed for each finger image.

After the setting of the printable area 309, the setting data selectedby a user from the setting data display area 302 is automaticallyreflected in the printable area 309.

It is possible for a user to freely change the size of the printablearea 309. Further, it is also possible for a user to delete the setprintable area 309. There may be a case where the fingernails onto whicha user desires to print the nail art are the fingernails of all thefingers of the hands or a case where they are the fingernails of part ofthe fingers. Because of this, in the present embodiment, it is madepossible for a user to set the desired printable area 309. Further, inthe present embodiment, as the printing target on which an image isreflected, the fingernail of a user is supposed, and therefore, there isa possibility that the size of the set printable area 309 is differentfrom one another. Consequently, in the present embodiment, it ispossible to print nail art at the appropriate position of a fingernailby a user inserting his/her hand into the printer 151 and setting thesize or the like of the printable area 309 while checking the actualimage of the fingernail. The printable area 309 that is set once may befollowed in the subsequent image recognition processing and the like.For example, in a case where the position of the finger (or fingernail)changes within the printer 151 after a user sets the printable area 309,the printable area 309 may change automatically following the image areaof the set printable area 309.

The print button 310 on the print data creation screen 301 is a buttonfor giving instructions to start printing. In a case where the printbutton 310 is pressed down, the nail application creates the print datafor performing printing on the fingernail based on the setting dataselected by a user from the setting data display area 302, which isreflected in the printable area 309. Then, the nail applicationtransmits the created print data to the printer 151 displayed in theprinter name display area 305. In the case shown in FIG. 3, the printdata is created by extracting the four printable areas 309 from thepreview area 307. The printer 151 performs printing on the fingernailbased on the print data sent from the nail application.

<Configuration of Information Processing Apparatus>

FIG. 6 is a function block diagram showing the software configuration ofthe information processing apparatus 101 for implementing the functionsdescribed previously. The information processing apparatus 101 has anail application 600. By the CPU 103 of the information processingapparatus 101 executing the program of the nail application 600, whichis stored in the ROM 104 or the external storage device 106, it is madepossible to cause the CPU 103 to perform each function of the nailapplication 600.

The nail application 600 has a display control unit 610. The displaycontrol unit 610 has a function to display the GUI screens including thenail art data selection screen 201 shown in FIG. 2, the nail image datasetting screen 501 shown in FIG. 5, and the print data creation screen301 shown in FIG. 3 on the display unit 108. Further, the displaycontrol unit 610 has the function of the input interface 102 andperforms various kinds of control in accordance with the operationinstructions that are input by a user.

The display control unit 610 has a nail art data selection receptionunit 611, a nail image data setting unit 612, an image capturinginstruction unit 613, an image display control unit 614, a fingernaildetection unit 615, a printable area setting unit 616, a reflection datareception unit 617, a reflection execution unit 618, and a printinginstruction unit 619.

The nail art data selection reception unit 611 receives a selection ofthe nail art data 203 from a user and acquires the selected nail artdata 203 as shown in FIG. 2. The nail image data setting unit 612acquires the nail image data that is set by a user in the fingernailareas 508 in the left hand preview area 503 and the right hand previewarea 504 as shown in FIG. 5. The image capturing instruction unit 613detects the pressing down of the Image capture button 306 in FIG. 3 orthe transition to the print data creation screen 301 in the state wherethe printer 151 is registered and instructs the printer 151 to capture amoving image in accordance with the detection. The image display controlunit 614 controls the display in the preview area 307 based on the imagedata, such as the moving image transmitted from the printer 151. Thefingernail detection unit 615 performs fingernail detection and outputsthe printable area 309 based on results of the fingernail detection. Theprintable area setting unit 616 sets the printable area 309 on thepreview area 307 in accordance with instructions from a user. Thereflection data reception unit 617 receives the setting data selected bya user from the setting data display area 302 and associates theselected setting data and the printable area 309 with each other. Thereflection execution unit 618 reflects the setting data selected by auser from the setting data display area 302 in the correspondingprintable area 309. The printing instruction unit 619 creates print datafor the printer 151 to perform printing based on the setting datareflected in the printable area 309 and transmits the created print datato the printer 151.

<Flow of Processing>

FIG. 7A and FIG. 7B indicate a sequence diagram showing an example ofthe processing in the present embodiment. The processing of the nailapplication 600 in FIG. 7A and FIG. 7B is performed by the CPU 103 ofthe information processing apparatus 101 loading the program code storedin the ROM 104 or the external storage device 106 onto the RAM 105 andexecuting the program code. Alternatively, it may also be possible toperform part or all of the processing in FIG. 7A and FIG. 7B byhardware, such as an ASIC and an electronic circuit. Symbol “S” in thefollowing explanation means each step in the sequence. In the following,by using FIG. 7A and FIG. 7B, the flow is explained in which a useractivates the nail application 600 and under the control of the nailapplication 600, the print engine 155 prints a nail art on thefingernail inserted into the printing target insertion unit 158.

First, a user activates the nail application 600. At S701, the displaycontrol unit 610 causes the display unit 108 to display the nail artdata selection screen 201 shown in FIG. 2. At S702, a user selects onepiece or a plurality of pieces of the nail art data 203 the user desiresto print on the nail art data selection screen 201. The nail art dataselection reception unit 611 selects the selection of one piece or aplurality of pieces of the nail art data 203 by a user via the nail artdata display area 202 on the nail art data selection screen 201. In acase where a user presses down the Determine button 204 after selectingone piece or a plurality of pieces of the nail art data 203, the displaycontrol unit 610 detects the pressing down and causes the display unit108 to display the nail image data setting screen 501 shown in FIG. 5.

In the nail art data display area 502 on the nail image data settingscreen 501, the one piece or the plurality of pieces of the nail artdata 203 selected by a user at S702 is displayed. At S703, a userselects one piece of nail art data from the nail art data displayed inthe nail art data display area 502. At S704, the nail image data settingunit 612 sets the nail image data included in the nail art data selectedat S703 to the fingernail area 508.

In a case where a user presses down the setting completion button 507after that, the display control unit 610 detects the pressing down andcauses the display unit 108 to display the print data creation screen301 shown in FIG. 3.

In the setting data display area 302 on the print data creation screen301 at this time, the setting data, which is the one piece or theplurality of pieces of nail image data set at S704, is displayed.

In a case where the printer 151 is not registered, the nail application600 performs search processing to search for a printer that cancommunicate with the nail application 600. After performing the searchprocessing, the display control unit 610 causes the display unit 108 todisplay a list of printers that can communicate with the nailapplication 600 as the results of the search. In a case where the listof printers is displayed on the display unit 108, at S705, a userdesignates a printer to use from the list. Here, it is assumed that auser designates the printer 151. Then, the display control unit 610receives instructions to designate the printer 151 by a user.

At S706, image capturing instruction unit 613 performs communicationprocessing for the selected printer 151 and instructs the printer 151 tocapture a moving image by the image capturing unit 157. At S706, it mayalso be possible for the image capturing instruction unit 613 totransmit a camera activation job to the printer 151 and for the printer151 to start moving image capturing by activating the image capturingunit 157 in response to the reception of the camera activation job.

At S707, the printer 151 transmits the moving image being captured bythe image capturing unit 157 to the information processing apparatus101. This moving image is displayed in the preview area 307 on the printdata creation screen 301 shown in FIG. 3. The moving image captured bythe image capturing unit 157 is displayed in the preview area 307 almostreal time. At this point in time, the state is such that the movingimage obtained by capturing the fingertip of a user, which is insertedinto the printing target insertion unit 158, is displayed in the previewarea 307.

At S708, a user selects one piece of setting data that is printed on thefingernail from the setting data displayed in the setting data displayarea 302. At S709, the nail art data selection reception unit 611receives instructions to select one piece of setting data by a user. Thedisplay control unit 610 highlights the setting data selected by a user.

At S710 a user inserts his/her hand into the printing target insertionunit 158 provided in the printer 151 that performs printing. At thistime, in order to print the nail image data that is set at S708 morevividly, a gel-like liquid is applied in advance to the fingernail of auser. In the following, this liquid is called base coat. A plurality ofcolors of the base coat exists, for example, white, translucent, and thelike. Further, it may also be possible for the image display controlunit 614 to display a message to prompt a user to insert his/her handinto the printing target insertion unit 158 of the printer at timingafter S709.

At S711, a user sets the printable area 309 in the state where themoving image transmitted at S707 is displayed in the preview area 307.Specifically, a user presses down the printable area setting button 308.

In fingernail detection at S716, to be described later, in response topressing down of the printable area setting button 308, the printablearea 309 is automatically displayed on the preview area 307 by thefingernail detection unit 615. In the present embodiment, it is assumedthat the number of the printable areas 309 that can be set is specifiedin advance to a predetermined value and for example, the case shown inFIG. 3 corresponds to the number of fingernails of the index finger, themiddle finger, the third finger, and the little finger of one hand, andtherefore, the predetermined value is specified as “4”. Thispredetermined value corresponds to the number of rails mounted in theprinting target insertion unit 158 and the number of pieces of nailimage data included in the setting data. In the following, explanationis given by taking an aspect as an example in which a user sets theprintable areas 309 corresponding to the same number as thepredetermined value, that is, an aspect in which a user sets the fourprintable areas 309. The number of the printable areas 309 set here isstored in the RAM 105 as a first predetermined value. That is, in thisexample, the first predetermined value is “4”.

At S712, the image capturing instruction unit 613 sends instructions tocapture a still image to the printer 151. In the image capturinginstructions that are sent to the printer 151, the rail numbercorresponding to the image capturing-target fingernail is included. Forexample, in a case where numbers 1 to 4 are assigned to the four railsin order from left, at the time of capturing the little finger of theleft hand, the number of “1” is sent, at the time of capturing the indexfinger of the left hand, the number of “4” is sent, at the time ofcapturing the index finger of the right hand, the number of “1” is sent,and at the time of capturing the little finger of the right hand, thenumber of “4” is sent.

At S713, the printer 151 performs adjustment by lifting the rail so thatthe height of the fingernail becomes the height in proximity to theprinter head of the printer 151. By adjusting the height of thefingernail to the height in proximity to the print head, it is madepossible for, in a case of an ink jet printer, the ink ejected from theprint head to land onto the fingernail securely. For the heightadjustment, a movable laser sensor mounted in the printer 151 is used.The laser sensor is installed at the height in proximity to the printhead and the fingernail is raised to the limit height at which the laseris not shut off. In the fingernail height adjustment, it is possible toperform the height adjustment more efficiently by narrowing the movablerange of the laser sensor to the fingernail range after grasping theapproximate position of the fingernail than taking the entire finger asthe movable range. Because of this, it is recommended to performfingernail detection in advance before performing the fingernail heightadjustment and send the fingernail position information from the nailapplication 600 to the printer 151 along with the image capturinginstructions at S712. In the present embodiment, the fingernaildetection in advance such as this is called “preliminary fingernaildetection”. In a case where the preliminary fingernail detection isperformed, the image capturing instructions are transmitted from thenail application 600 to the printer 151 before S712. In the preliminaryfingernail detection, the fingernail height adjustment is not performed,and therefore, it is not necessary to include the rail number describedpreviously in the image capturing instructions to be transmitted for thepreliminary fingernail detection. The printer 151 having received theimage capturing instructions performs image capturing of a still imagein the image capturing unit 157. In a case where the image capturing ofthe still image is completed, the printer 151 transmits the capturedimage to the information processing apparatus 101. The nail application600 detects the fingernail from the received captured image and storesinformation (referred to as fingernail position information) indicatingthe positions of all the detected fingernails. Then, the nailapplication 600 transmits the fingernail position information on theimage capturing-target fingernails to the printer 151 along with therail numbers at the time of giving image capturing instructions at S712.The printer 151 adjusts the fingernail height by narrowing the movablerange of the laser sensor using the received fingernail positioninformation.

In a case where the fingernail height adjustment is completed, at S714,the image capturing unit 157 performs image capturing of the stillimage. In a case where the image capturing is completed, the heights ofall the rails are returned to the initial values and at S715, theprinter 151 transmits the image capturing data to the informationprocessing apparatus 101.

At S716, the nail application 600 detects the fingernail area from thereceived captured image. For the fingernail area detected at this step,the printable area 309 on the print data creation screen 301 is set. Asone method of fingernail detection, there is a method of detecting thewhite color of the base coat applied to the fingernail by imageprocessing. Specifically, by using the RGB values of each pixel in thecaptured image, the pixel whose pixel values exceeds predeterminedthreshold values (for example, R>200, G>200, B>200) is detected and thedetected pixel is determined to be the pixel in the fingernail area. Inorder to prevent the rail under the finger from being detectederroneously as a fingernail at the time of fingernail detection, it isrecommended to set the color of the rail to a color other than white,such as black. Further, in order to prevent the area in which thecaptured image becomes the white area due to the reflection of lightfrom being detected erroneously, it is desirable to configure the railfrom a material that reflects light diffusedly. Further, in thedetection by image processing, detection becomes difficult in a casewhere the base coat is translucent, and therefore, it may also bepossible to use machine learning as another detection method. By usingfinger images including fingernails to which the white or translucentbase coat is applied as the images that are learned in machine learning,it is made possible to detect the fingernail area even in a case of thetranslucent base coat, not only in a case of the white base coat. Inmachine learning, a learning model is constructed by causing thelearning model to learn where the fingernail exists in a preparedlearning image. The constructed learning model is incorporated in thenail application 600 and processed by the CPU 103 or the GPU 110 andthen made use of to detect the fingernail area from the captured image.The color of skin and the shape of fingernail are different fordifferent persons, and therefore, by preparing patterns of many hands aslearning images and causing the learning model to learn them, it is madepossible to detect the fingernail of any user. In the machine learning,a number of frameworks exist, and therefore, by making use ofalready-existing frameworks, it is possible to implement machinelearning.

The processing at S712 to S716 is performed repeatedly the number oftimes corresponding to the number of image capturing-target fingernails.For example, in a case where the four fingernails of the little fingerto the index finger of the left hand are a target, image capturing,transmission of image capturing data, and finger detection are performedrepeatedly for each finger. In this case, a total of four capturedimages are obtained, but at the time of the printable area setting unit616 setting the printable area 309, the portion in which the targetfingernail is captured is cut out from each captured image and displayedin the preview area 307. That is, in the preview area 307, four fingerimages cut out from the four captured images are displayed side by sideand the printable area 309 is set to each fingernail image.

After the printable area 309 is set by the printable area setting unit616, at S717, the reflection data reception unit 617 associates the nailimage data that is set as the setting data and the printable area 309that is set with each other. Then, the reflection execution unit 618reflects the setting data in the printable area 309 based on thisassociation. In the case shown in FIG. 3, association is performedbetween each of the four kinds of nail image data included in thesetting data 302(c) and each of the four printable areas 309 that areset.

At S718, a user edits the printable area 309 as needed. Specifically, itis possible for a user to adjust the size or the like of the printablearea 309 and increase in number the printable area 309. At this time,the reception of editing instructions is performed by the reflectiondata reception unit 617 and the reflection of the edited results isperformed by the reflection execution unit 618. In the editing at thisstep, it is also possible to adjust which portion of the nail art in thenail image data is reflected for the printable area, not only to adjustthe size of the printable area.

At S719, a user presses down the print button 310. In response to thepressing down of the print button 310, the printing instruction unit 619creates print data for printing an image in which the setting dataselected by a user from the setting data displayed in the setting datadisplay area 302 is reflected in the printable area 309. To explain thisby taking a specific example, after reflecting the setting data selectedby a user from the setting data displayed in the setting data displayarea 302 in the printable area 309 and checking the display contents,the user presses down the print button 310. The printing instructionunit 619 creates print data in response to the pressing down of theprint button 310. At this time, it may also be possible for the nailapplication 600 to display a message to the effect that “Do not moveyour hand inserted into the printing target insertion unit 158”. As theprint data, it is necessary to create print data whose image size is thesame as the printable range of the printer 151 and correctly arrange theprintable area 309 so that the printing position does not deviate.Consequently, in order to specify the printable range of the printer151, a red frame indicating the printable range is provided in theprinter 151 and the printable area is specified by detecting the redframe by image processing from the image captured in the image capturingunit 157. At the time of detecting the red frame, the red frame isdetected from the outside of the captured image. By detecting the redframe from the outside as described above, even in a case where the redpainting invades the inside of the red frame, it is possible toaccurately detect the red frame. By arranging the printable area 309 inthe print data based on the detected printable range, it is madepossible to perform printing at the correct position.

At S720, the image capturing instruction unit 613 sends instructions tocapture a still image to the printer 151. The image capturing inaccordance with the instructions that are sent at this step is imagecapturing that is performed in order to determine whether a deviation inthe fingernail position has occurred during the period from thefingernail detection at S716 until the print button 310 is pressed downat S719. As at S712, in the image capturing instructions that are sentto the printer 151 at this step, the rail number corresponding to theimage capturing-target fingernail is included.

At S721, the printer 151 lifts the rail to the same height as that atthe time of lifting the rail at S713. Due to this, the fingernailposition is set again to the position suitable for printing.

At S722, the image capturing unit 157 performs image capturing of astill image.

At S723, the printer 151 transmits image capturing data to theinformation processing apparatus 101.

At S724, the nail application 600 performs fingernail detection. Thefingernail detection performed at this step is performed by the samemethod as that of the fingernail detection performed at S716.

At S725, the printing instruction unit 619 corrects the print data sothat the print data fits to the fingernail area detected at S724. Theprint data correction processing at this step will be described later byusing FIG. 8A to FIG. 8C and FIG. 9.

At S726, the nail application 600 transmits the print data corrected atS725 to the printer 151.

The above processing at S720 to S726 is the processing relating to theprint data correction, but in a case where a user edits the position orsize of the printable area 309 at S718, there is a possibility that thefingernail area has not been detected correctly in the fingernaildetection, and therefore, the processing relating to the print datacorrection is skipped. In that case, the processing at S720, S722 toS725 is skipped and at S726, the print data created at S719 istransmitted without correction in place of the print data corrected atS725. Further, in a case where the fingernail detection has failed atS724, there is a possibility that the hand has been pulled out from theprinting target insertion unit 158, and therefore, the nail application600 displays an error message to the effect that printing is aborted onthe display unit 108 and skips the processing at the subsequent steps.

At S727, the printer 151 performs printing based on the transmittedprint data. As a result of this step, the nail art represented by thenail image data is printed on the fingernail of a user.

The processing at S720 to S727 is performed the number of timescorresponding to the number of printing-target fingernails. For example,in a case where the four fingernails of the little finger to the indexfinger of the left hand are a target, image capturing, transmission ofimage capturing data, fingernail detection, transmission of print data,and printing are performed repeatedly for each finger. In a case wherethe printer 151 performs printing from the bottom of the fingernailtoward the tip of the fingernail, the movement distance of the printhead is shorter by performing printing in order of the little finger,the third finger, the middle finger, the index finger than that byperforming printing in order of the index finger, the middle finger, thethird finger, and the little finger because of the arrangementrelationship of the fingers. On the contrary, in a case where theprinter 151 performs printing from the tip of the fingernail toward thebottom of the fingernail, the movement distance of the print head isshorter by performing printing in order of the index finger, the middlefinger, the third finger, and the little finger than that by performingprinting in order of the little finger, the third finger, the middlefinger, and the index finger. As described above, the order of thefingers to be printed is changed by taking into consideration that thefinger arrangement is different depending on the printing-target hand(left hand/right hand). Due to this, it is possible to reduce themovement distance of the print head, resulting in a reduction in theprinting time.

The printer 151 continues fingernail image capturing also duringprinting and in a case where it is detected that the hand is pulled outfrom the printing target insertion unit 158 of the printer 151 beforeprinting is completed, it may be possible to terminate printing.Further, it may also be possible for the nail application 600 to detectthat the hand is pulled out from the printing target insertion unit 158of the printer 151 before printing is completed and give instructions toterminate printing to the printer 151.

<Correction Processing of Print Data>

FIG. 8A to FIG. 8C are diagrams showing the outline of the print datacorrection processing at S725. Specifically, FIG. 8A to FIG. 8C showcorrection examples of a print image in a case where there is a changein the position, the gradation, and the shape of the fingernail based onthe comparison between the results of the fingernail detection at S716and the results of the fingernail detection at S724.

FIG. 8A shows an example in a case where the fingernail position changesbetween fingernail detection results 801 at S716 and fingernaildetection results 802 at S724. In a case where the fingernail positionchanges, the printing instruction unit 619 acquires a corrected printimage 804 by adjusting the position of the nail art in a print image 803created at S719, that is, by moving the position by the distance bywhich the fingernail position deviates. By correcting the print image asdescribed above, it is possible to perform printing by adjusting theposition to the fingernail position immediately before the printing evenin a case where the fingernail position changes during the period fromthe detection of the fingernail at S716 until the printing is started.

FIG. 8B shows an example in a case where the fingernail gradient changesbetween fingernail detection results 805 at S716 and fingernaildetection results 806 at S724. In a case where the fingernail gradientchanges, the printing instruction unit 619 acquires a corrected printimage 808 by adjusting the position of the nail art in a print image 807created at S719, that is, by rotating the position by the amount bywhich the fingernail gradient changes. By correcting the print image asdescribed above, it is possible to perform printing by adjusting thegradient to the fingernail gradient immediately before the printing evenin a case where the fingernail gradient changes during the period fromthe detection of the fingernail at S716 until the printing is started.

FIG. 8C shows an example in a case where the fingernail shape changesbetween fingernail detection results 809 at S716 and fingernaildetection results 810 at S724. In a case where the fingernail shapechanges, the printing instruction unit 619 acquires a corrected printimage 812 by adjusting the shape of the nail art in a print image 811created at S719, that is, by changing the shape so as to fit to thechanged fingernail shape. By correcting the print image as describedabove, it is possible to perform printing by adjusting the shape to thefingernail shape immediately before the printing even in a case wherethe fingernail shape changes during the period from the detection of thefingernail at S716 until the printing is started. As a case where thefingernail shape changes, mention is made of a case where the fingernailshape captured by the camera of the image capturing unit 157 changesbecause the finger is bent or the like during the period from thedetection of the fingernail until the printing is started, and the like.

There is a possibility that the position, the gradient, and the shape ofthe fingernail change respectively, and therefore, the correctionprocessing is performed by combining each piece of the correctionprocessing in FIG. 8A to FIG. 8C. For example, in a case where theposition and the gradient of the fingernail change, the printinginstruction unit 619 performs both the correction processing in FIG. 8Aand the correction processing in FIG. 8B.

Further, in FIG. 8A to FIG. 8C, the examples are shown in which theprint image created in advance is corrected, but it may also be possiblefor the printing instruction unit 619 to create a print image anew byreflecting the nail image data in the area detected in the fingernaildetection (fingernail detection immediately before printing) at S724. Inthis case, it is possible to simplify the processing because thecorrection processing is no longer necessary, but the arrangementresults of the nail art data for the printable area 309, which areacquired by the editing at S718, are no longer reflected in the printimage correctly.

FIG. 9 is a flowchart of the print data correction processing. At S901,the printing instruction unit 619 compares the fingernail detectionresults at S716 and the fingernail detection results at S724 and detectsa change in the position, the gradient, and the shape of the fingernail,respectively. For example, in a case of the fingernail position, thedistance between the center positions of the fingernails in the twofingernail detection results is derived. Further, in a case of thegradient, the difference in the angle between the fingernails is derivedand in a case of the shape, the difference in the size between therectangles surrounding the fingernails is derived.

Following the above, at S902, the printing instruction unit 619determines whether the amount of change is larger than or equal to apredetermined threshold value for each of the position, the gradient,and the shape of the fingernail. In a case where the amount of change ineven one of the position, the gradient, and the shape is larger than orequal to the predetermined threshold value, the determination results atthis step will be affirmative. In a case where the determination resultsat this step are affirmative, the processing advances to S904 and on theother hand, in a case where the determination results are negative, theprocessing advances to S903. For example, in a case where thepredetermined threshold value for the fingernail position is set to 3mm, whether the distance between the center positions derived at S901 islonger than or equal to 3 mm is determined. Similarly, in a case wherethe threshold value for the gradient is set to ten degrees, whether thedifference in the angle derived at S901 is larger than or equal to tendegrees is determined. For the threshold value of the shape, a thresholdvalue is set to the vertical width and the horizontal width,respectively, of the rectangle surrounding the fingernail and whetherthere is a change larger than or equal to the value is determined.

In a case where the determination results at S902 are affirmative, it isregarded that the fingernail has moved more than a permitted range ofcorrection, and at S904, the printing instruction unit 619 causes thedisplay unit 108 to display an error message to the effect that printingis aborted. In this case, it is not possible to perform the subsequentprinting processing, and therefore, the subsequent printing processingis skipped.

In a case where the determination results at S902 are negative, at S903,the printing instruction unit 619 corrects the print data in accordancewith the change in the position, the gradient, or the shape of thefingernail. The correction processing at this step is as explained inFIG. 8A to FIG. 8C.

Effects of the Present Embodiment

As explained above, in the present embodiment, the print data iscorrected in accordance with the change in the position and the like ofthe fingernail immediately before starting printing. Due to this, evenin a case where the fingernail position or the like changes during theperiod from the image capturing for creating print data untilimmediately before the start of printing, it is possible to correctlyprint the nail art on the fingernail. For example, even in a case wherethe fingernail is inclined, it is possible to print the nail art data onthe fingernail in the correct orientation.

Other Embodiments

In the embodiment described previously, explanation is given by takingthe printer 151 as an example, whose aspect is such that one hand isinserted into the printing target insertion unit 158, but the presentdisclosure is not limited to this aspect. For example, the aspect of theprinter may be one, such as a printer that is installed in a shop, inwhich both hands are inserted into the printing target insertion unit.In this case, it may be possible for an employee in the shop to performthe GUI operation of the nail application.

Further, in the embodiment described previously, explanation is give bytaking the case as an example where the image (pattern) is printedmainly as a nail art, but an aspect may also be accepted in which astructure including patterns is formed as a nail art by using shape dataand image data representing a three-dimensional structure and the like.

Further, in the embodiment described previously, the aspect is explainedin which a hand is inserted into the printing target insertion unit 158of the printer 151 and printing is performed directly on the fingernail,but another aspect may be accepted. For example, it is possible to applythe concept of the embodiment described previously in a case whereprinting is performed by using a printer that performs printing for anobject that is attached to the fingernail, for example, such a seal.

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

According to one embodiment of the present invention, in a case wherethe fingernail position or the like changes during the period from thedetection of a printing-target fingernail area until printing isperformed, it is made possible to perform printing correctly.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2020-130541, filed Jul. 31, 2020, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A non-transitory computer readable storage mediumstoring a program for causing a computer of an information processingapparatus to function as: a first detection unit configured to detect aprinting-target fingernail area from a first captured image; a creationunit configured to create print data for performing printing in afingernail area detected by the first detection unit; a second detectionunit configured to detect a printing-target fingernail area from asecond captured image before transmitting the print data to a printer;and a correction unit configured to correct the print data based on adifference between first detection results by the first detection unitand second detection results by the second detection unit; a thirddetection unit configured to detect a change in position, gradient, andshape of a fingernail based on the first detection results and thesecond detection results; and a determination unit configured todetermine whether one of an amount of change in position, an amount ofchange in gradient, and an amount of change in shape of a fingernail islarger than or equal to a predetermined threshold value.
 2. The storagemedium according to claim 1, wherein the correction unit corrects theprint data in accordance with each of the changes in a case wheredetermination results by the determination unit are negative.
 3. Thestorage medium according to claim 1, storing a program for causing acomputer of the information processing apparatus to further function as:a display control unit configured to display an error message to theeffect that printing is aborted on a GUI screen in a case wheredetermination results by the determination unit are affirmative.
 4. Thestorage medium according to claim 1, storing a program for causing acomputer of the information processing apparatus to further function as:a display control unit configured to display an error message to theeffect that printing is aborted on a GUI screen in a case where thesecond detection unit cannot detect a fingernail area.
 5. The storagemedium according to claim 1, wherein in a case where the print data isedited by a user, detection by the second detection unit and correctionof the print data by the correction unit are not performed.
 6. Thestorage medium according to claim 1, storing a program for causing acomputer of the information processing apparatus to further function as:a first transmission unit configured to transmit image capturinginstructions for acquiring a first captured image to the printer; and asecond transmission unit configured to transmit image capturinginstructions for acquiring a second captured image to the printer.
 7. Aninformation processing apparatus comprising: at least one processorcausing the information processing apparatus to act as: a firstdetection unit configured to detect a printing-target fingernail areafrom a first captured image; a creation unit configured to create printdata for performing printing in a fingernail area detected by the firstdetection unit; a second detection unit configured to detect aprinting-target fingernail area from a second captured image beforetransmitting the print data to a printer; and a correction unitconfigured to correct the print data based on a difference between firstdetection results by the first detection unit and second detectionresults by the second detection unit; a third detection unit configuredto detect a change in position, gradient, and shape of a fingernailbased on the first detection results and the second detection results;and a determination unit configured to determine whether one of anamount of change in position, an amount of change in gradient, and anamount of change in shape of a fingernail is larger than or equal to apredetermined threshold value.
 8. A control method of an informationprocessing apparatus, the control method comprising: a first detectionstep of detecting a printing-target fingernail area from a firstcaptured image; a creation step of creating print data for performingprinting in a fingernail area detected at the first detection step; asecond detection step of detecting a printing-target fingernail areafrom a second captured image before transmitting the print data to aprinter; and a correction step of correcting the print data based on adifference between first detection results at the first detection stepand second detection results at the second detection step; a thirddetection step of detecting a change in position, gradient, and shape ofa fingernail based on the first detection results and the seconddetection results; and a determination step of determining whether oneof an amount of change in position, an amount of change in gradient, andan amount of change in shape of a fingernail is larger than or equal toa predetermined threshold value.